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Abstract. We consider Markov Decision Processes (MDPs) with mean-payoff 
parity and energy parity objectives. In system design, the parity objective is used 
to encode cj-regular specifications, and the mean-payoff and energy objectives 
can be used to model quantitative resource constraints. The energy condition re- 
quires that the resource level never drops below 0, and the mean-payoff condi- 
tion requires that the limit-average value of the resource consumption is within 
a threshold. While these two (energy and mean-payoff) classical conditions are 
equivalent for two-player games, we show that they differ for MDPs. We show 
that the problem of deciding whether a state is almost-sure winning (i.e., winning 
with probability 1) in energy parity MDPs is in NP n coNP, while for mean- 
payoff parity MDPs, the problem is solvable in polynomial time, improving a 
recent PSPACE bound. 



1 Introduction 

Markov decision processes (MDPs) are a standard model for systems that exhibit both 
stochastic and nondeterministic behaviour. The nondeterminism represents the freedom 
of choice of control actions, while the probabilities describe the uncertainty in the re- 
sponse of the system to control actions. The control problem for MDPs asks whether 
there exists a strategy (or policy) to select control actions in order to achieve a certain 
goal with a certain probability. MDPs have been used in several areas such as planning, 
probabilistic reactive programs, verification and synthesis of (concurrent) probabilistic 
systems [12,24, 1]. 

The control problem may specify a goal as a set of desired traces (such as w-regular 
specifications), or as a quantitative optimization objective for a payoff function on the 
traces of the MDP Typically, discounted-payoff and mean-payoff functions have been 
studied [15]. Recently, the energy objectives (corresponding to total-payoff functions) 
have been considered in the design of resource-constrained embedded systems [3, 21, 7] 
such as power-limited systems, as well as in queueing processes, and gambling models 
(see also [4] and references therein). The energy objective requires that the sum of the 
rewards be always nonnegative along a trace. Energy objective can be expressed in the 
setting of boundary less one-counter MDPs [4]. In the case of MDPs, achieving energy 
objective with probability 1 is equivalent to achieving energy objective in the stronger 
setting of a two-player game where the probabilistic choices are replaced by adversarial 
choice. This is because if a trace p violates the energy condition in the game, then a finite 
prefix of p would have a negative energy, and this finite prefix has positive probability in 



the MDP. Note that in the case of two-player games, the energy objective is equivalent 
to enforce nonnegative mean-payoff value [3, 5]. 

In this paper, we consider MDPs equipped with the combination of a parity objective 
(which is a canonical way to express the cj-regular conditions [23]), and a quantitative 
objective specified as either mean-payoff or energy condition. Special cases of the parity 
objective include reachability and fairness objectives such as Biichi and coBiichi condi- 
tions. Such combination of quantitative and qualitative objectives is crucial in the design 
of reactive systems with both resource constraints and functional requirements [6, 11, 3, 
2]. For example, Kucera and Strazvosky consider the combination of PCTL with mean- 
payoff objectives for MDPs and present an EXPTIME algorithm [19]. In the case of 
energy parity condition, it can also be viewed as a natural extension of boundaryless 
one-counter MDPs with fairness conditions. 

Consider the MDP in Fig. 1, with the objective to visit the Biichi state q 2 infinitely 
often, while maintaining the energy level (i.e., the sum of the transition weights) non- 
negative. A winning strategy from go would loop 20 times on qo to accumulate en- 
ergy and then it can afford to reach the probabilistic state from which the Biichi state 
is reached with probability % and cost 20. If the Biichi state is not reached immedi- 
ately, then the strategy needs to recharge 10 units of energy and try again. This strategy 
uses memory and it is also winning with probability 1 for the nonnegative mean-payoff 
Biichi objective. In general however, the energy and mean-payoff parity objectives do 
not coincide (see later the example in Fig. 2). In particular, the memory requirement 
for energy parity objective is finite (at most exponential) while it may be infinite for 
mean-payoff parity. 

We study the computational complexity of the problem of deciding if there exists a 
strategy to achieve energy parity objective, or mean-payoff parity objective with proba- 
bility 1 (i.e., almost-surely). We provide tight bounds for this problems in the following 
sense. 

1 . For energy parity MDPs, we show that the problem is in NP n coNP, and present 
a pseudo-polynomial time algorithm. Our bounds are the best conceivable upper 
bound unless parity games can be solved in P 3 , which is a long-standing open ques- 
tion. 

2. For mean-payoff parity MDPs, we show that the problem is solvable in polynomial 
time (and thus PTIME-complete). Our result improves the recent PSPACE upper 
bound of [17] for this problem. 

We refer to [12, 16, 10] for importance of the computation of almost-sure winning set 
related to robust solutions (independence of precise transition probabilities) and the 
more general quantitative problem. The computation of the almost-sure winning set 
in MDPs typically relies either on the end-component analysis, or analysis of attrac- 
tors and sub-MDPs. The result of [17] for mean-payoff parity MDPs uses the analysis 
of attractors and sub-MDPs and gives a nested recursive PSPACE algorithm for the 
problem similar to the classical algorithm for parity games. Our results rely on the 
end-component analysis, but in a much more refined way than the standard analysis, to 
obtain a polynomial-time algorithm. Our proof combines techniques for mean-payoff 



3 Parity games polynomially reduce to two-player energy games [18,3,5], and thus to energy 
MDPs. Hence the problem for almost-sure energy parity MDPs is at least as hard as solving 
two player parity games. 
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Fig. 1. An energy Biichi MDP. The player- 1 states are qo, qi, and the probabilistic state is qi. 

and parity objectives to produce infinite-memory strategy witnesses, which is necessary 
in general. We present an algorithm that iterates successively over even priorities 2i and 
computes almost-sure winning end-components with the even priority 2i as the best 
priority. 

For energy parity MDPs the end-component based analysis towards polynomial- 
time algorithm does not work since solving energy parity MDPs is at least as hard 
as solving two-player parity games. Instead, for energy parity MDPs, we present a 
quadratic reduction to two-player energy Biichi games which are in NP n coNP and 
solvable in pseudo-polynomial time [7]. 

Due to lack of space we present the detailed proofs in the appendix. 

2 Definitions 

Probability distributions. A probability distribution over a finite set A is a function 

k : A — > [0, 1] such that J2 a eA K ( a ) = The support of n is the set Supp(«;) = {a G 
A | n(a) > 0}. We denote by V{A) the set of probability distributions on A. 

Markov Decision Processes. A Markov Decision Process (MDP) M = (Q,E,5) 
consists of a finite set Q of states partitioned into player-1 states Qi and probabilistic 
states Q P (i.e., Q = Qi U Qp), a set E C Q x Q of edges such that for all q G Q, 
there exists (at least one) q' G Q such that (q, q') G E, and a probabilistic transition 
function 5 : Qp — > T>(Q) such that for all q G Qp and q 1 G Q, we have (q, q 1 ) G E 
iff 5{q){q') > 0. We often write S(q, q') for S(q)(q'). For a state q G Q, we denote by 
E{q) = {q' G Q \ (q, q') G E} the set of possible successors of q. 

End-components and Markov chains. A set U C Q is S-closed if for all q e U n Qp 

we have Supp(<5(<7)) C U. The sub-MDP induced by a <5-closed set U is M \ U = 
(U, E n (U x U), S). Note that M \ U is an MDP if for all q G U there exists q' G U 
such that (q, q') G E. A Markov chain is a special case of MDP where Q\ = 0. A 
closed recurrent set for a Markov chain is a (5-closed set U C Q which is strongly 
connected. End-components in MDPs play a role equivalent to closed recurrent sets in 
Markov chains. Given an MDP M = (Q, E, 8) with partition (Qi, Qp), a set U C Q 
of states is an end- component if {/ is <5-closed and the sub-MDP M |" U is strongly 
connected [12, 13]. We denote by £ (M) the set of end-components of an MDP M. 

Plays. An MDP can be viewed as the arena of a game played for infinitely many rounds 
from a state qo G Q as follows. If the game is in a player-1 state q, then player 1 chooses 
the successor state in the set E(q); otherwise the game is in a probabilistic state q, and 
the successor is chosen according to the probability distribution 8(q). This game results 
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in a play from q , i.e., an infinite path p = g <Zi • • • such that ((ft, ft+i) £ -E for all 
i > 0. The prefix of length n of p is denoted by p(n) = q n . . . q n , the last state of p(n) 
is Last(/?(n)) = q n . We write for the set of all plays. 

Strategies. A strategy (for player 1) is a function a : Q*Q\ — \ T>(Q) such that for 
all p £ Q*, q £ Qi, and q' £ Q P , if a(p ■ q)(q') > 0, then (q, q') £ E. We denote 
by S the set of all strategies. An outcome of a from qo is a play qo<Zi • • • where qi + \ £ 
Supp(cr(<7o . . . qi)) for alii > such that q^ £ Q\. 

Outcomes and measures. Once a starting state q £ Q and a strategy a £ £ are fixed, 
the outcome of the game is a random walk uj^ for which the probabilities of every event 
A C Q, which is a measurable set of plays, are uniquely defined [24] . For a state q £ Q 
and an event A C Q, we denote by P£ (^4) the probability that a play belongs to A if 
the game starts from the state q and player 1 follows the strategy a. For a measurable 
function / : Q — > M we denote by [/] the expectation of the function / under the 
probability measure P£ (•)■ 

Strategies that do not use randomization are called pure. A player- 1 strategy a is 
pure if for all p £ Q* and q £ Q\, there is a state q' £ Q such that cr(p • q)(q') = 1. 

Finite-memory strategies. A strategy uses finite -memory if it can be encoded by a 
deterministic transducer (Mem, mo,a u , a n ) where Mem is a finite set (the memory of 
the strategy), mo <E Mem is the initial memory value, a u : Mem x Q — > Mem is 
an update function, and a n : Mem x Qi — > T>(Q) is a next-move function. The size 
of the strategy is the number |Mem| of memory values. If the game is in a player-1 
state q, and m is the current memory value, then the strategy chooses the next state 
q' according to the probability distribution a n (m,q), and the memory is updated to 
a u (m,q). Formally, (Mem, m , a u , a n ) defines the strategy a such that <r(p ■ q) = 
Qi>(a«(mo, p), q) for a H P 6 Q* an d Q £ Q\, where a u extends a u to sequences 
of states as expected. A strategy is memoryless if |Mem| = 1. For a finite-memory 
strategy a, let M a be the Markov chain obtained as the product of M with the transducer 
defining a, where ((m,q), (m',q')) is an edge in M a if m' — a u (m,q) and either 
q £ Qi and q' £ Supp(a„(m, q)), or q £ Q P and {q, q') £ E. 

Two-player games. A two-player game is a graph G = (Q, E) with the same assump- 
tions as for MDP, except that the partition of Q is denoted (Qi, Q2) where Q2 is the set 
of player-2 states. The notions of play, strategies (in particular strategies for player 2), 
and outcome are analogous to the case of MDP [7]. 

Objectives. An objective for an MDP M (or game G) is a set cj> C Q of infinite 
paths. Let p : Q — > N be a priority function and w : E — > 7L be a weight function' 
where positive numbers represent rewards. We denote by W the largest weight (in ab- 
solute value) according to w. The energy level of a prefix 7 = q qi . . . q n of a play is 
EL(ui, 7) = 2^™=o 1 w ( ( lii Qi+i)> an d tne niean-payoff value 5 of a play p = go<Zi • • • is 
MP(w, p) = liminfn^oo i • EL(w, p(n)). In the sequel, when the weight function w; 
is clear from the context we omit it and simply write EL(7) and MP(p). We denote by 

4 Sometimes we take the freedom to use rational weights (i.e., w : E — > Q), while we always 
assume that weights are integers encoded in binary for complexity results. 

5 The results of this paper hold for the definition of mean-payoff value using lim sup instead of 
lim inf. 
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Inf(p) the set of states that occur infinitely often in p, and we consider the following 
objectives: 

- Parity objectives. The parity objective Parity(p) = {p G ft \ mm{p(q) \ q G 
lnf(p)} is even } requires that the minimum priority visited infinitely often be even. 
The special cases of Bilchi and coBiichi objectives correspond to the case with two 
priorities, p : Q — > {0, 1} andp : Q — V {1, 2} respectively. 

- Energy objectives. Given an initial credit c G N, the energy objective 
PosEnergy(c ) = {p G ft | Vn > : c + EL(p(n)) > 0} requires that the 
energy level be always positive. 

- Mean-payoff objectives. Given a threshold v G Q, the mean-payoff objective 
MeanPayoff-^ = {p G J? | MP(p) > v] (resp. Mean Payoff >!/ = {p G 1? | 
M P (p) > ^}) requires that the mean-payoff value be at least v (resp. strictly greater 
than v). 

- Combined objectives. The energy parity objective Parity(p) n PosEnergy(co) and 
the mean-payoff parity objective Parity(p) n M ean Payoff ~ v (for ~G {>,>}) com- 
bine the requirements of parity and energy (resp., mean-payoff) objectives. 

Almost-sure winning strategies. For MDPs, we say that a player- 1 strategy a is 
almost-sure winning in a state q for an objective <fi if ^q(4>) — 1- F° r two-player games, 
we say that a player- 1 strategy a is winning in a state q for an objective <fi if all outcomes 
of a starting in q belong to (f>. For energy objectives with unspecified initial credit, we 
also say that a strategy is (almost-sure) winning if it is (almost-sure) winning for some 
finite initial credit. 

Decision problems. We are interested in the following problems. Given an MDP M 
with weight function w and priority function p, and a state qo, 

- the energy parity problem asks whether there exists a finite initial credit Co G N 
and an almost-sure winning strategy for the energy parity objective from qo with 
initial credit cq. We are also interested in computing the minimum initial credit 
in q which is the least value of initial credit for which there exists an almost-sure 
winning strategy for player 1 in q . A strategy for player 1 is optimal in qo if it is 
winning from q with the minimum initial credit; 

- the mean-payoff parity problem asks whether there exists an almost-sure winning 
strategy for the mean-payoff parity objective with threshold from q . Note that it 
is not restrictive to consider mean-payoff objectives with threshold because for 
^G {>, >}, we have MP(w, p) ~ v iff MP(u> — u, p) ~ 0, where w — v is the 
weight function that assigns w(e) — v to each edge e € E. 

The two-player game versions of these problems are defined analogously [7]. It is 
known that the initial credit problem for simple two-player energy games [6, 3], as well 
as for two-player parity games [14] can be solved in NP n coNP because memoryless 
strategies are sufficient to win. Moreover, parity games reduce in polynomial time to 
mean-payoff games [18], which are log-space equivalent to energy games [3,5]. It is 
a long-standing open question to know if a polynomial-time algorithm exists for these 
problems. Finally, energy parity games and mean-payoff parity games are solvable in 
NP D coNP although winning strategies may require exponential and infinite memory 
respectively, even in one-player games (and thus also in MDPs) [11,7]. 

The decision problem for MDPs with parity objective, as well as with mean-payoff 
objective, can be solved in polynomial time [15, 12, 10, 13]. However, the problem is 
in NP n coNP for MDPs with energy objective because an MDP with energy objective 
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Fig. 2. The gadget construction is wrong for mean-payoff parity MDPs. Player 1 is almost-sure 
winning for mean-payoff Biichi in the MDP (on the left) but player 1 is losing in the two-player 
game (on the right) because player 2 (box-player) can force a negative-energy cycle. 



is equivalent to a two-player energy game (where the probabilistic states are controlled 
by player 2). Indeed (1) a winning strategy in the game is trivially almost-sure winning 
in the MDP, and (2) if an almost-sure winning strategy a in the MDP was not winning 
in the game, then for all initial credit c there would exist an outcome p of a such that 
Co + EL(p(i)) < for some position i > 0. The prefix p(i) has a positive probability in 
the MDP, in contradiction with the fact that a is almost-sure winning. As a consequence, 
solving MDP with energy objectives is at least as hard as solving parity games. 

In this paper, we show that the decision problem for MDPs with energy parity ob- 
jective is in NP n coNP, which is the best conceivable upper bound unless parity games 
can be solved in P. And for MDPs with mean-payoff parity objective, we show that 
the decision problem can be solved in polynomial time, improving a recent PSPACE 
bound [17]. 

The MDP in Fig. 2 on the left, which is essentially a Markov chain, is an exam- 
ple where the mean-payoff parity condition is satisfied almost-surely, while the energy 
parity condition is not, no matter the value of the initial credit. For initial credit c , the 
energy will drop below with positive probability, namely * +1 . 

End-component lemma. We now present an important lemma about end-components 
from [12, 13] that we use in the proofs of our result. It states that for arbitrary strategies 
(memoryless or not), with probability 1 the set of states visited infinitely often along a 
play is an end-component. This lemma allows us to derive conclusions on the (infinite) 
set of plays in an MDP by analyzing the (finite) set of end-components in the MDP. 

Lemma 1. [12, 13] Given an MDP M, for all states q g Q and all strategies a e S, 
we have PJ({cj | lnf(w) e £(M)}) = 1. 

3 MDPs with Energy Parity Objectives 

We show that energy parity MDPs can be solved in NP n coNP, using a reduction to 
two-player energy Biichi games. Our reduction also preserves the value of the minimum 
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Fig. 3. Gadget for probabilistic states in energy Biichi MDP. Diamonds are probabilistic states, 
circles are player 1 states, and boxes are player 2 states. 

initial credit. Therefore, we obtain a pseudo-polynomial algorithm for this problem, 
which also computes the minimum initial credit. Moreover, we show that the memory 
requirement for almost-sure winning strategies is at most 2-|Q|-W, which is essentially 
optimal 6 . 

We first establish the results for the special case of energy Biichi MDPs. We present 
a reduction of the energy Biichi problem for MDPs to the energy Biichi problem for 
two-player games. The result then follows from the fact that the latter problem is in 
NP n coNP and solvable in pseudo-polynomial time [7]. 

Given an MDP M, we can assume without loss of generality that every probabilis- 
tic state has priority 1, and has two outgoing transitions with probability \ each [25, 
Section 6]. We construct a two-player game G by replacing every probabilistic state of 
M by a gadget as in Fig. 3. The probabilistic states q of M are mapped to player-2 
states in G with two successors (q, L) and (q, R). Intuitively, player 2 chooses (q, L) to 
check whether player 1 can enforce the Biichi condition almost-surely. This is the case 
if player 1 can reach a Biichi state (with priority 0) infinitely often when he controls 
the probabilistic states (otherwise, no Biichi state is ever visited, and since (•, L) states 
have priority 1, the Biichi condition is not realized in G). And player 2 chooses (q, R) 
to check that the energy condition is satisfied. If player 2 can exhaust the energy level 
in G, then the corresponding play prefix has positive probability in M. Note that (q, R) 
has priority and thus cannot be used by player 2 to spoil the Biichi condition. 

Formally, given M = (Q, E, 6) with partition (Qi,Qp) of Q, we construct a game 
G = (Q', E') with partition (Q'^Q'p) where Q[ = Q 1 U (Q P x {L}) and Q' 2 = 
QpU (Qp x {R}), see also Fig. 3. The states in Q' that are already in Q get the same 
priority as in M, the states (•, L) have priority 1, and the states (•, R) have priority 0. 
The set E' contains the following edges: 

- all edges (q, q') e E such that q G Q\\ 

- edges (q, (q, d)), ((q, d), q') for all q g Q P , d g {L, R}, and q' g Supp(<5(<?)). 



6 Example 1 in [7] shows that memory of size 2- (|Q| — V)-W + 1 may be necessary. 
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The edges (q, q') and ((q, d),q') in E' get the same weight as (q, q') in M, and all edges 
(q, (q, d)) get weight 0. 

Lemma 2. Given an MDP M with energy Biichi objective, we can construct in linear 
time a two-player game G with energy Biichi objective such that for all states q n in M, 
there exists an almost-sure winning strategy from qo in M if and only if there exists a 
winning strategy from qo in G (with the same initial credit). 

Note that the reduction presented in the proof of Lemma 2 would not work for 
mean-payoff Biichi MDPs. Consider the MDP on Fig. 2 for which the gadget-based 
reduction to two-player games is shown on the right. The game is losing for player 1 
both for energy and mean-payoff parity, simply because player 2 can always choose to 
loop through the box states, thus realizing a negative energy and mean-payoff value (no 
matter the initial credit). However player 1 is almost-sure winning in the mean-payoff 
parity MDP (on the left in Fig. 2). 

While the reduction in the proof of Lemma 2 gives a game with n' — \Qi \ + 3 ■ 
\Qp\ states, the structure of the gadgets (see Fig. 3) is such that the energy level is 
independent of which of the transitions (q, (q, L)) or (q, (q, R)) is taken. Since from the 
result of [7, Lemma 8] and its proof, it follows that the memory updates in winning 
strategies for energy Biichi games can be done according to the energy level of the play 
prefix, it follows that the memory bound of 2-n-W can be transfered to almost-sure 
winning strategies in Energy Biichi MDPs, where n = |Win n Qi \ is the number of 
player 1 almost-sure winning states. Also, the pseudo-polynomial algorithm for solving 
two-player energy Biichi games can be used for MDPs, with the same 0(\E\-\Q\ 5 -W) 
complexity [7, Table 1] . 

Using Lemma 2, we solve energy parity MDPs by a reduction to energy Biichi 
MDPs. The key idea of the reduction is that if player 1 has an almost-sure winning 
strategy for the energy parity objective, then player 1 can choose an even priority 2i 
and decide to satisfy the energy objective along with satisfying that priority 2i is visited 
infinitely often, and priorities less than 2i are visited finitely often. 

W.l.o.g. we assume that player-1 states and probabilistic states alternate, i.e. E(q) C 
Qi for all q G Qp, and E(q) C Q P for all q G Q\. The reduction is then as follows. 
Given an MDP M = (Q,E,8) with a priority function p : Q — > N and a weight 
function w : E -> Z, we construct (M',p',w'} as follows. M' is the MDP M = 
(Q',E',S') where: 

- Q' = Q U (Q x {0, 2, . . . , 2r}) U {sink} where 2r is the largest even priority of 
a state in Q. Intuitively, a state (q, i) G Q' corresponds to the state q of M from 
which player 1 will ensure to visit priority i (which is even) infinitely often, and 
never visit priority smaller than i; 

- E' contains E U {(sink, sink)} and the following edges. For each probabilistic state 
q G Q P , fori = 0,2,...,2r, 

• (a) if p(q') > i for all q' G E(q), then ((q, i), (q', i)) G E 1 for all q' G E(q), 

• (b) otherwise, ((q, i), sink) G E'. 

For each player 1 state q G Qi, for each q 1 G E(q), for i = 0, 2, . . . , 2r, 

• (a) (q, sink) G E' and ((q, i), sink) G E', and 

• (b) if p(q') > i, then {q, (q', i)) G E' and ((q, i), (q\ *)) £ E'. 
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The partition (Q'^Q'p) of Q' is defined by Q[ = Q 1 U (Qi x {0, 2, . . . , 2r}) U {sink} 
and Q' P = Q' \ Q[. The weight of the edges (q, q'), (q,(q',i)) and ((q, i), (q' , i)) 
according to w' is the same as the weight of (q, q') according to w. The states (q, i) 
such that p(q) = i have priority according to p' (they are the Biichi states), and all the 
other states in Q' (including sink) have priority 1. 

Lemma 3. Given an MDP M with energy parity objective, we can construct in 
quadratic time an MDP M' with energy Biichi objective such that for all states qo 
in M, there exists an almost-sure winning strategy from qo in M if and only if there 
exists an almost-sure winning strategy from q in M' (with the same initial credit). 

From the proof of Lemma 3, it follows that the memory requirement is the same as 
for energy Biichi MDPs. And if the weights are in {-1, 0, 1}, it follows that the energy 
parity problem can be solved in polynomial time. 

Theorem 1. For energy parity MDPs, 

1. the decision problem of whether a given state is almost-sure winning is in 
NP n coNP, and there is a pseudo-polynomial time algorithm in O ( | E \ ■ d ■ \ Q | 5 • W) 
to solve it; 

2. memory of size 2-\Q\-W is sufficient for almost-sure winning strategies. 

4 MDPs with Mean-payoff Parity Objectives 

In this section we present a polynomial-time algorithm for solving MDPs with mean- 
payoff parity objective. We first recall some useful properties of MDPs. 

For an end-component U G £{M), consider the memoryless strategy ojj that plays 
in every state s G U H Qi all edges in E(s) n U uniformly at random. Given the 
strategy o\j, the end-component U is a closed connected recurrent set in the Markov 
chain obtained by fixing ojj . 

Lemma 4. Given an MDP M and an end-component U G £(M), the strategy o\j 
ensures that for all states s G U, we have F° u ({w | lnf(w) = U}) = 1. 

Expected mean-payoff value. Given an MDP M with a weight function w, the ex- 
pected mean-payoff value, denoted ValMP(w), is the function that assigns to every 
state the maximal expectation of the mean-payoff objective that can be guaranteed by 
any strategy. Formally, for q G Q we have Va\MP(w)(q) = sup aeE (MP(to)), 
where MP(w) is the measurable function that assigns to a play p the long-run av- 
erage MP(w,p) of the weights. By the classical results of MDPs with mean-payoff 
objectives, it follows that there exists pure memoryless optimal strategies [15], i.e., 
there exists a pure memoryless optimal strategy a* such that for all q G Q we have 
ValMPHO?) = Ef (MP(to)). 

It follows from Lemma 4 that the strategy ujj ensures that from any starting state s, 
any other state t is reached in finite time with probability 1. Therefore, the value for 
mean-payoff parity objectives in MDPs can be obtained by computing values for end- 
components and then playing a strategy to maximize the expectation to reach the values 
of the end-components. 
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We now present the key lemma where we show that for an MDP that is an end- 
component such that the minimum priority is even, the mean-payoff parity objective 
Parity(p) n MeanPayoff-" is satisfied with probability 1 if the expected mean-payoff 
value is at least v at some state (the result also holds for strict inequality). In other 
words, from the expected mean-payoff value of at least v we ensure that both the mean- 
payoff and parity objective is satisfied with probability 1 from all states. The proof of 
the lemma considers two pure memoryless strategies: one for stochastic shortest path 
and the other for optimal expected mean-payoff value, and combines them to obtain an 
almost-sure winning strategy for the mean-payoff parity objective (details in appendix). 

Lemma 5. Consider an MDP M with state space Q, a priority function p, and weight 
function w such that (a) M is an end-component (i.e., Q is an end-component) and 
(b) the smallest priority in Q is even. If there is a state q 6 Q such that ValMP(w) > v 
( resp. ValM P(w) > v), then there exists a strategy a* such that for all states q G Q we 
haveVf (Parity(p)nMeanPayoff- 1 ') = l(resp. Pf (Parity(p)nMeanPayoff >!/ ) = 1). 

Memory required by strategies. Lemma 5 shows that if the smallest priority in an end- 
component is even, then considering the sub-game restricted to the end-component, the 
mean-payoff parity objective is satisfied if and only if the mean-payoff objective is 
satisfied. The strategy constructed in Lemma 5 requires infinite memory, and in the 
case of loose inequality (i.e., MeanPayoff-' y ) infinite memory is required in general 
(see [11] for an example on graphs), and if the inequality is strict (i.e., MeanPayoff >!/ ), 
then finite memory strategies exist [17]. For the purpose of computation we show that 
both strict and non-strict inequality can be solved in polynomial time. Since Lemma 5 
holds for both strict and non-strict inequality, in sequel of this section we consider non- 
strict inequality and all the results hold for non-strict inequality as well. 

Winning end-component. Given an MDP M with a parity objective Parity(p) 
and a mean-payoff objective Mean Payoff-" for a weight function w, we call an 
end-component U winning if (a) min(p([/)) is even; and (b) there exists a state 
with expected mean-payoff value at least v in the sub-MDP induced by U, i.e., 
maxqgy ValMP(w)(q) > v in the sub-MDP induced by U. We denote by W the set 
of winning end-components, and let Win = \J UeW U be the union of the winning 
end-components. 

Reduction to reachability of winning end-component. By Lemma 5 it follows that in 
every winning end-component the mean-payoff parity objective is satisfied with prob- 
ability 1. Conversely, consider an end-component U that is not winning, then either 
the smallest priority is odd, or the maximal expected mean-payoff value that can be 
ensured for any state in U by staying in U is less than v. Hence if only states in U 
are visited infinitely often, then with probability 1 (i) either the parity objective is not 
satisfied, or (ii) the mean-payoff objective is not satisfied. In other words, if an end- 
component that is not winning is visited infinitely often, then the mean-payoff parity 
objective is satisfied with probability 0. It follows that the value function for MDPs 
with mean-payoff parity objective can be computed by computing the value function for 
reachability to the set Win, i.e., formally, sup^g^ P^(Parity(p) n MeanPayoff-") = 
sup^g^ Pg (Reach(Win)), where Reach(Win) is the set of paths that reaches a state in 
Win at least once. Since the value function in MDPs with reachability objectives can 
be computed in polynomial time using linear programming [15], it suffices to present a 
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polynomial-time algorithm to compute Win in order to obtain a polynomial-time algo- 
rithm for MDPs with mean-payoff parity objectives. 

Computing winning end-components. The computation of the winning end- 
components is done iteratively by computing winning end-components with smallest 
priority 0, then winning end-components with smallest priority 2, and so on. The com- 
putation of Win is as follows: 
- For i > 0, let W2i be the set of maximal end-components U with states with priority 
at least 2i and that contain at least one state with priority 2i, i.e., U contains only 
states with priority at least 2i, and contains at least one state with priority 2i. Let 
y^2i ^= be the set of maximal end-components U E W2i such that there is a 
state q E U such that the expected mean-payoff value in the sub-MDP restricted to 
U is at least v. Let Win2i = {J UeW > U. 

The set Win = U^ 2- ' ^' n2i ' s ^ e un i° n °f tne states of the winning end-components 
(formal pseudo-code in the appendix). 

Complexity of computing winning end-components. The winning end-component 
algorithm runs for 0(d) iterations and in each iteration requires to compute a maxi- 
mal end-component decomposition and compute mean-payoff values of at most n end- 
components, where n is the number of states of the MDP. The maximal end-component 
decomposition can be achieved in polynomial time [12, 13, 10]. The mean-payoff value 
function of an MDP can also be computed in polynomial time using linear program- 
ming [15]. It follows that the value function of an MDP with mean-payoff parity ob- 
jectives can be computed in polynomial time. The almost-sure winning set is obtained 
by computing almost-sure reachability to Win in polynomial time [12, 13, 10]. This 
polynomial-time complexity provides a tight upper bound for the problem, and closes 
the gap left by the PS PACE upper bound of [17]. 

Theorem 2. The following assertions hold: 

1. The set of almost-sure winning states for mean-payoff parity objectives can be com- 
puted in polynomial time for MDPs. 

2. For mean-payoff parity objectives, almost-sure winning strategies require infinite 
memory in general for non-strict inequality (i.e, for mean-payoff parity objectives 
Parity(p) fl M ean Payoff- v ) and finite -memory almost-sure winning strategies exist 
for strict inequality (i.e., for Parity(p) n M ea n Payoff >v ). 

Discussion. We considered MDPs with conjunction of mean-payoff parity and energy 
parity objectives, and presented tight complexity bounds, algorithms, and bounds for the 
memory required by strategies. The disjunction of mean-payoff parity and energy parity 
objectives are straightforward and summarized in Theorem 3 (details in appendix). 

Theorem 3. The following assertions hold: 

1. The set of almost- sure winning states for disjunction of mean-payoff and parity 
objectives can be computed in polynomial time for MDPs. 

2. The decision problem of whether a given state is almost-sure winning for disjunc- 
tion of energy and parity objectives is in NP fl coNP for MDPs. 
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Appendix 



5 Details of Section 3 

Proof (of Lemma 2). We show that player 1 has an almost-sure winning strategy in M 
if and only if player 1 has a winning strategy in the game G (for the same initial credit). 

First, we show that if player 1 has an almost-sure winning strategy gm in M, then 
we can construct a winning strategy g g in G. We can assume that gm is pure [9, The- 
orem 5]. 

To define gq, we assign a rank to prefixes of outcomes of gm in M as follows. 
Prefixes p such that p(Last(p)) = get rank 0. For other prefixes p (with p(Last(p)) = 
1), if Last(p) e Qi is a player-1 state, then p gets rank 1 + rank(p • q) where q is 
such that a(p)(q) = 1; if Last(p) 6 Qp is a probabilistic state, then p gets rank 1 + 
min{rank( ( o') | p 1 is a ranked successor of p}. Prefixes without ranked successor get 
no rank. We claim that all prefixes p compatible with gm get a (finite) rank. Otherwise, 
there would exist a non-ranked prefix compatible with gm (thus reachable with positive 
probability) such that all its extensions are unranked. This would imply that only states 
with priority 1 are visited from that point on, hence the co-Biichi objective has positive 
probability, in contradiction with the fact that gm is almost-sure winning for energy 
Biichi. 

We construct the pure strategy gg as follows. Given a play prefix pa in G, let h(p G ) 
be the sequence obtained from pa by deleting all states of the form (q, d) for q 6 Qp 
and d e {L, R}. Note that h(p G ) is a play in M. Let q G — V.zst(p G ) e Q[, we define 
cg(pg) as follows: 

- if qG G Qu then G G (p G ) = G M (h(p G )); 

- if qa = (q, L) (for q e Q P ), then G G (p G ) = q' where rank(/9 G • q') < rank(p G ). 

Note that for every outcome p G of g g , the play h(p G ) is an outcome of gm in M. To- 
wards contradiction, assume that g g is not winning in G. Then, there exists an outcome 
p G of g g that violates either: 

- the energy condition; then, the energy level drops below after finitely many steps 
in pc, and this occurs as well in h(p G ) with positive probability in M, a contradic- 
tion with the fact that g m is almost-sure winning for energy Biichi in M. 

- or the Biichi condition; then, from some point on in p G only priority 1 is visited. 
This implies that in the gadgets, eventually only (q, L) states are visited. Then, ac- 
cording to the definition of gq, the rank in prefixes of p G decreases and eventually 
reaches rank 0, that is a state with priority is visited, and we have again a contra- 
diction. 

Therefore, g g is a winning strategy in the game G. 

Second, we show that if player 1 has a winning strategy g g in G, then we can 
construct an almost-sure winning strategy gm in M. By the result of [7, Lemma 8] and 
its proof, we can assume that g g is energy-based memoryless, that is G G (p) = G G (p') 
for all p,p' such that Last(p) = Last(p') and EL(p) = EL(p'). In particular, if h(p) = 
h(p'), then G G (p) = G G (p'). 

We define the strategy gm as follows: for each prefix pu in M, let gm(pm) — 
G G (p) where p is such that h(p) = pu- By the above remark, the strategy gm is 
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uniquely and well denned. We also know that og uses finite memory. Therefore, in G aG 
all cycles are have nonnegative energy and visit a priority state. Therefore, all cycles 
in M aM have nonnegative energy; and if there is a reachable closed recurrent set U in 
M„ M that contains only priority 1 states, then in G UG player 2 can fix a strategy to 
reach the closed recurrent set U (by choosing the successor of probabilistic states using 
(•, R) states) and in the states of U, player 2 always chooses (•, L) states. The (unique) 
outcome is a play that eventually remains in the closed recurrent set and therefore visits 
priority 1 states only from some point on, spoiling strategy uq, a contradiction. Hence, 
all closed recurrent sets in M aM contain a priority state and the Biichi objective is 
satisfied with probability 1 under strategy <jm- □ 

Proof (of Lemma 3). Consider the construction of (M',p', w') defined before Lemma 3. 
Let Win' C Q x {0, 2, . . . , 2r} be the set of almost-sure winning states in M' for the 
energy Biichi objective and let Win — {q G Q \ 3i-(q,2i) G Win'} be the projection of 
Win' on Q. We then convert all states in W to absorbing (or sink) states with weight 0, 
and then consider almost-sure energy Biichi winning set Z with Win as the Biichi set 
(this is almost-sure energy and reachability to Win). 

We claim Z is the almost-sure winning set for energy parity in M. The proof is as 
follows. Let Z = Q\Z. Consider an arbitrary strategy a for player 1 and a starting state 
q G Z. Assume towards contradiction that a is almost-sure winning for energy parity 
objective. Suppose there is an end-component U such that U R Z ^ 0, that is visited 
infinitely often with positive probability. Since a is almost-sure winning, we must have 
that min(p([/)) is even (say 2i) and the energy objective is satisfied. Hence in the copy 
2i in M' we have that U is almost-sure winning. This means U x {2i} C Win' and since 
U C Q we have U C Win. But this contradicts that UriZ ^ and Win C Z. It follows 
that there is no end-component that intersects with Z that is visited infinitely often with 
positive probability. Hence, given a, the set Z must be reached with probability 1. If 
the energy objective is also ensured with probability 1 by a, then the strategy is almost- 
sure winning for energy and reachability to Win (since from Z almost-sure winning 
for energy and reachability to Win can be ensured). This shows that q would belong to 
Z. This is a contradiction and completes the proof. □ 

Bound for strategies. We construct an almost-sure winning strategy of size at most 
2 • (\Z\ + 1) • W as follows. We first partition the set Win as follows: Win is the set of 
states that is winning in copy 0; Wini is the set of states that is winning in copy 2 and 
not in copy 0; and so on. For a state q G Win, let q G Win 2 i, then for the state q we 
play the almost-sure winning strategy for in copy 2i. Since the copies are disjoint, the 
total memory required for the almost-sure winning strategies is J^. 2 • | Win^i\ ■ W = 
2 ■ | Win | • W. For states q G Z \ Win, we play the almost-sure winning strategy to 
reach Win ensuring the energy objective. Since for the reachability to Win we can 
consider states in Win as a single absorbing state, the memory required is at most 
2 • (\Z \ Win\ + 1) • W. After reaching Win the strategy switches to the almost- 
sure winning strategy from Win. Hence the total memory required by the almost-sure 
winning strategy is at most 2 • (\Z\ + 1) • W. 

Algorithm. If we simply apply the algorithm for energy Biichi MDPs on the reduction, 
then we obtain a 0(\E \ ■ d ■ (d ■ |Q|) 5 • W) algorithm. The improved version is obtain by 
simply following the steps of the proof. First, for each copy we compute the almost-sure 
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winning set for the energy Biichi objective Since each copy is disjoint and in each copy 
we require 0(|£7| • |Q| 5 • W), the total time required to compute the Win is at most 
0(\E\-d-\Q\ a ■ W). Finally the almost-sure energy reachability to Win can be achieved 
in an additional 0(\E\ ■ \Q\ 5 ■ W) time. Hence we obtain an 0(\E\ ■ d ■ |Q| 5 • W) time 
algorithm. 

6 Details of Section 4 

Proof (of Lemma 5). The strategy a* for the mean-payoff parity objective is produced 
by combining two pure memoryless strategies: a m for the expected mean-payoff ob- 
jective and oq for the objective of reaching the smallest priority. We present a few 
properties that we use in the correctness proof of the almost-sure winning strategy. 

1. Property 1. Finite-time reach to smallest priority. Observe that under the strategy 
<7q we obtain a Markov chain such that every closed recurrent set in the Markov 
chain contains states with the smallest priority, and hence from all states q a state 
with the smallest priority (which is even) is reached in finite time with probability 1 . 

2. Property 2. Uniform value. The expected mean-payoff value for all states q e Q 
is the same: if we fix the memoryless strategy a u that chooses all successors uni- 
formly at random, then we get a Markov chain as the whole set Q as a closed 
recurrent set, and hence from all states q 6 Q any state q' G Q is reached in fi- 
nite time with probability 1, and hence the expected mean-payoff value at q is at 
least the expected mean-payoff value at q'. It follows that for all q,q' G Q the ex- 
pected mean-payoff value at q and q' coincide. Let us denote the uniform expected 
mean-payoff value by v* . 

3. Property 3. Property of optimal mean-payoff strategy. The strategy u m is a pure 
memoryless strategy and once it is fixed we obtain a Markov chain. The limit of 
the average frequency (or Cesaro limit) exists for all states and since a m is optimal 
it follows that for all states q G Q we have 

1 n 

lim - .yX»»M(0 i) i+1 ))]=t;*, 

i=l 

where 0, is the random variable for the i-th state of a path. In the resulting Markov 
chain obtained by fixing a m , the expected mean-payoff value for every closed re- 
current set must be v*; otherwise, if there is a closed recurrent set with expected 
mean-payoff value less than v*, then there must be a closed recurrent set with ex- 
pected mean-payoff value greater than v* as all states have the uniform value v*, 
but then we obtain a state with expected mean-payoff value greater than v* which 
contradicts Property 2. Hence from the theory of finite state Markov chains (the 
almost-sure convergence to the Cesaro limit [20]) we obtain that 

V°({p\Km - e -EL(w,p(i)) >v*})= limP;({p | - ( -EL(w, p(£)) > v*}) = 1. 

In the above equality the limit and the probability operators are exchanged using 
Lesbegue's Dominated Convergence Theorem [22] (as the weights are bounded). 
Hence for all e > 0, there exists j(e) G N such that if a m is played for any I > j(e) 
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steps then the average of the weights for I steps is at least e within the expected 
mean-payoff value of the MDP with probability at least 1 — e, i.e., for all q 6 Q, 
for all I > j(e) we have 

F°({p\j.EL(w,p(£))>v*-e})>l-e. 

Let W be the maximum absolute value of the weights. The almost-sure strategy a* 
for mean-payoff parity objective is played in rounds, and the strategy for round i is as 
follows: 

1. Stage 1. First play the strategy ctq till the smallest priority is reached. 

2. Stage 2. Let e t = 1/i. If the game was in the first stage in this (i-th round) for ki 
steps, then play the strategy a m for li steps such that £j > max{j(ei); i ■ ki ■ W}. 
This ensures that the with probability at least 1 — ej the average of the weights in 
round i is at least 

ii ■ (v* - ei) - h ■ W _ {li +h)-v* -ii-a-ki-v* -h-W 



> v* - 



h + e t 

e t + kj ■ v* + h ■ W 
&i H~ ki 



> v — ei — (since v < W) 

1% ki 



>v* 



2 -h-W 

li 



2 

> v* — 6i (since li > i ■ k ■ W) 

i 

= v*-- 

i 

Then the strategy proceeds to round i + 

The strategy ensures that there are infinitely many rounds (this follows by Property 1 of 
finite-time reachability to min even priority state). Hence with probability 1 the smallest 
priority that is visited infinitely often is the smallest priority of the end-component 
(which is even). This ensures that the parity objective is satisfied with probability 1. 
We now argue that the mean-payoff objective is also satisfied with probability 1 . Fix 
arbitrary e > and consider i such that ? < e. For all j > i, in round j, the average 
weights is at least v* — e with probability at least 1 — e. Since mean-payoff objective is 
independent of finite prefixes, for all q e Q we have 

P£({p| lim l.ELKp(*))>«*-e})>l-e. 
Since e > is arbitrary, letting e — >• 0, we obtain that for all q 6 Q we have 

F°({p\ lim - ■ EL( W , PW) >v*})>1 

H £-5-00 I 

Hence depending on whether v* > v or v* > v we obtain the desired result. □ 
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Algorithm 1: AlgoWinEndComponent 

Input: An MDP M with parity function p, weight function w and threshold v. 
Output: The set W of union of winning end-components. 
l.i := 0; 

2. M := M; 

3. Fori := to |d/2j do 

3.1 Compute the maximal end-component decomposition of M;; 

3.2 Let Wn be the maximal end-components U 

such that U C Uj>2»P _1 (j) and [/ n p" 1 (2i) / 0; 

3.3 Let C W2i be the set of maximal end-components U 6 W2; such that 

in the sub-MDP induced by to U there exists q with Va/( Mean Payoff (w)) > v. 

3.4 w 2i := u uewi( (7; 

3.5 Z2i := Random attractor of W21 in M<; 

3.6 Afj+i := sub-MDP induced by removing Z^i in M»; 

3.7 i := i+ 1; 

4. return W := U}5 2J W 2i . 



Further details about computing winning end-components for MDPs with mean- 
payoff parity objectives. We now present some further details about computing the 
winning end-components with mean-payoff parity objectives. The computation of the 
winning end-components is done iteratively by computing winning end-components 
with smallest priority 0, then winning end-components with smallest priority 2, and 
so on. We start with the initial MDP Af := M. In iteration i the remaining MDP is 
Mi. We compute the maximal end-component decomposition of Mj, then consider the 
maximal-end components U that contains only states with priority at least 2i, and at 
least one state with priority 2i. If there is such an end component U where the expected 
mean-payoff value is at least v at some state, then U is included in Wz%. The we consider 
the random attractor (i.e., alternating reachability to Wn by the random player) to Wn 
and the set of random attractor is removed from the MDP for the next iteration. The 
random attractor to a set T is as follows: T := T and for i > we have T i+i := 
T t U {q e Qi I W e Q.(q,q') e E -> q' GTi}U{q G Q P \ 3q' e Q.(q,q') £ 
E A q' G Ti] and the random attractor is lj i>0 Tj. It follows from the results of [10] 
(see Lemma 2.1 of [10]) that if we consider a set of end-components, and take random 
attractor to the set, then the maximal end-component decomposition of the remaining 
MDP remains unaffected. Moreover, the complement of a random attractor in an MDP 
is always a sub-MDP. The set W — Ui=o^ is the union of the states of the winning 
end-components. The formal pseudocode is given as Algorithm 1. 

7 Details of Theorem 3 

We first consider the disjunction of mean-payoff and parity objectives, and then dis- 
junction of energy and parity objectives. 

Disjunction of mean-payoff and parity objectives. For the disjunction of mean- 
payoff and parity objectives in MDPs we consider end-components analysis. An end- 
component is winning if either the parity objective can be ensured almost-surely, or 
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the mean-payoff objective can be ensured almost-surely. Since determining almost-sure 
winning for parity objective and mean-payoff objectives can be done in polynomial 
time, we can use the algorithm of Section 4 for computing winning end-components 
and then reachability to winning end-components. Hence disjunction of mean-payoff 
parity objectives can be solved in polynomial time, and also pure memoryless optimal 
strategies exist. 

Disjunction of energy and parity objectives. The solution of disjunction of energy and 
parity objectives is achieved using the end-component analysis: an end-component is 
winning if either the parity objective can be ensured almost-surely or the energy objec- 
tive can be ensured almost-surely. Whether an end-component is almost-sure winning 
for parity can be decided in polynomial time, and for energy objectives it is in NP D 
coNP. Hence the winning end-components can be determined in NP n coNP. Let W\ be 
the union of the set of winning end-components for almost-sure parity, and let W2 be the 
union of the set of remaining winning end-components (i.e., only almost-sure winning 
for energy). Finally we need to ensure almost-sure reachability to W\ or almost-sure 
energy reachability to W2. Again this can be achieved in NP n coNP. 
The desired result follows (also see discussion section in [8]). 
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